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The findings from this review do not reflect the full body of research evidence on charter schools. 


What is this study about? 

The study examined the effect of charter schools 
on annual student achievement growth in reading 
and math in 25 states, the District of Columbia, 
and New York City. The study primarily used data 
on students in grades 3-8, but additional elemen- 
tary and high school grades were included for 
several states. 

Researchers analyzed data from state-level stan- 
dardized reading and math tests administered dur- 
ing the 2008-09 through 2010-11 school years. The 
study authors also updated a previous analysis that 
had been conducted on a subset of 16 states with 
additional years of data. 

More than 1 .5 million charter school students were 
matched to students attending traditional public 
schools based on test scores and demographic 
characteristics. Eighty-five percent of charter school 
students were successfully matched. 

To measure the impact of charter schools on annual 
achievement growth, study authors compared the 
year-to-year test score changes of charter school 
students with those of matched students attending 
traditional public schools. 


Features of Charter Schools 


Charter schools are public schools that are 
established on the basis of a contract, or charter, 
that a private board holds. They are exempt from 
many state and district regulations that govern 
traditional public schools, including those involving 
staffing, curriculum, and budget decisions. 


WWC Rating 


The research described in this 
report meets WWC evidence 
standards with reservations 

This study was a large, multi-year analysis in which 
the authors matched charter school students with 
traditional public school students based on observed 
demographic characteristics and test scores. How- 
ever, unobserved differences between the two groups 
may have existed. For example, charter school 
students may have been more motivated to do well in 
school or may have had other unobserved character- 
istics that influenced student achievement. 

In addition, the study’s results do not have a straight- 
forward interpretation because they blend the 1-year 
gains students experienced during their first year of 
charter school attendance and 1-year gains during 
subsequent years. 

Finally, the effect sizes reported in this study (which 
are based on an analysis of achievement gains) are 
not directly comparable to effect sizes reported by 
other studies that analyzed achievement levels. 


What did the study find? 

The authors reported that charter school students 
in the sample had annual reading score growth that 
was 0.01 standard deviations higher than that of 
students in traditional public schools. This difference 
was statistically significant. There was no statisti- 
cally significant difference between charter school 
students and traditional public school students in 
their year-to-year gains in math. 
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Appendix A: Study details 

Center for Research on Education Outcomes (CREDO). (2013). National Charter School Study: 2013. 

Stanford, CA: Author. 

Setting The study analyzed data gathered from students who attended charter schools and traditional 
public schools in 25 states, the District of Columbia, and New York City. The 25 states were: 
Arizona, Arkansas, California, Colorado, Florida, Georgia, Illinois, Indiana, Louisiana, Massa- 
chusetts, Michigan, Minnesota, Missouri, Nevada, New Jersey, New Mexico, New York, North 
Carolina, Ohio, Oregon, Pennsylvania, Rhode Island, Tennessee, Texas, and Utah. The authors 
refer to this as a “27-state” sample because they estimated impacts separately for New York 
City and for the state of New York excluding New York City, and they counted the District of 
Columbia as a state. In addition to impacts estimated for the 27-state sample, the authors 
estimated impacts for a subset of 16 states that were included in an earlier study (CREDO, 
2009. Multiple choice: Charter school performance in 16 states. Stanford University, CA). 2 The 
16 states in the earlier study were: Arizona, Arkansas, California, Colorado (Denver only), the 
District of Columbia, Florida, Georgia, Illinois (Chicago only), Louisiana, Massachusetts, Min- 
nesota, Missouri, New Mexico, North Carolina, Ohio, and Texas. 

Study sample For each state, the study used data for students in grades 1-1 1 who attended charter schools 
and traditional public schools during 5 school years (from 2006-07 to 2010-11), as available. 
The specific grades and school years available for each state differed, but all states had data 
for grades 3-8, and all states except New Jersey had data from the 2008-09 through 201 0-1 1 
school years. The study emphasized analyses based on data from the 2008-09 through 
2010-1 1 school years, so this review focuses on the corresponding findings. 

For each year of the study, the authors identified feeder schools— defined as traditional public 
schools that students in charter schools would have attended had the charter schools not 
existed. Each student in a given charter school was matched to one or more students from the 
relevant feeder school on race/ethnicity, gender, English proficiency, subsidized lunch status, 
special education status, and grade level. Exact matches were required on each of these vari- 
ables, and students in traditional public schools who attended a charter school in later years 
were excluded from being matched to a charter school student. Additional matching was done 
using the reading or math test score from the prior school year. The matching was conducted 
separately for reading and math test scores. To be considered a valid match, the prior year 
test scores of intervention and comparison students were required to differ by no more than 
0.10 standard deviations. Using this process, it was possible for a charter school student to 
be matched to more than one traditional public school student. A “virtual twin” record was 
created for each charter school student by averaging the outcomes of up to seven traditional 
public school students with whom the charter school student was matched. 

For the 27-state sample, valid matches were created for 86% of the tested charter school stu- 
dents in reading and 84% of the tested charter school students in math. The match percentages 
by state and subject range from a low of 73% in New Jersey for math to a high of 96% for Ten- 
nessee in reading. After conducting matches, the 27-state sample included more than 1 .5 million 
charter school students from more than 4,500 charter schools. The 16-state sample included 
nearly 2,500 charter schools whose students had been included in the earlier CREDO study. 
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Intervention 

group 

The intervention group included students who were enrolled in a charter school in a study 
state during the 2008-09 through 201 0-1 1 school years. Charter schools are public schools 
that are established on the basis of a contract, or charter, that a private board holds. Charter 
schools are typically intended to improve student learning and achievement by encourag- 
ing the use of different and innovative learning methods and establishing a new system of 
accountability for schools. Charter schools are exempt from many state and district regula- 
tions that govern traditional public schools, including those involving staffing, curriculum, and 
budget decisions. 

Comparison 

group 

The comparison group included students from traditional public schools that were feeder 
schools to the charter schools included in the study. Students from traditional public schools 
who subsequently attended a charter school were excluded from the comparison group. 

Outcomes and 
measurement 

The analyses that are the focus of this review examined year-to-year gains in standardized 
reading and math test scores from state-level assessments that were administered to students 
in the spring of each school year from 2008-09 through 201 0-1 1 . For a more detailed descrip- 
tion of these outcome measures, see Appendix B. 

Support for 
implementation 

Not applicable. 

Reason for 
review 

This study was identified for review by the WWC by receiving significant media attention. 
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Appendix B: Outcome measures for each domain 


General reading achievement 

Gain in standardized reading scores 

Reading assessment scores were obtained from state-administered assessments given to students in specific 
grades. These scores were transformed into standardized reading scores by the study authors so that the 
standardized scores had a mean of 0 and a standard deviation of 1 within a state, grade, and year. The gain in 
standardized reading scores for a student was calculated as the difference between the student’s standardized 
reading score in the current year and the standardized reading score in the previous year. 

General math achievement 

Gain in standardized math scores 

Math assessment scores were obtained from state-administered assessments given to students in specific 
grades. These scores were transformed into standardized math scores by the study authors so that the 
standardized scores had a mean of 0 and a standard deviation of 1 within a state, grade, and year. The gain in 
standardized reading scores for a student was calculated as the difference between the student’s standardized 
math score in the current year and the standardized math score in the previous year. 
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Appendix C: Study findings for each domain 





Mean 

(standard deviation) 

WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention Comparison 
group group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

General reading achievement 

Gain in standardized reading 
scores 

27 

states 

4,549 charter 
schools/ 

1,734,232 charter 
school students 

nr nr 

0.01 

0.01 

0 

<0.01 

Domain average for general reading achievement 



0.01 

0 

Statistically 

significant 

General math achievement 

Gain in standardized math 
scores 

27 

states 

4,547 charter 
schools/ 

1,672,167 charter 
school students 

nr nr 

-0.01 

-0.01 

0 

>0.05 

Domain average for general math achievement 



-0.01 

0 

Not 

statistically 

significant 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students 
who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the 
change in an average student's percentile rank that can be expected if the student is given the intervention. Because this study examined year-to-year gains in achievement, the 
effect sizes and improvement indices presented are not directly comparable to effect sizes that would be estimated in an analysis of achievement levels. The WWC-computed 
average effect size is a simple average rounded to two decimal places; the average improvement index is calculated from the average effect size. The statistical significance of the 
study’s domain average was determined by the WWC. nr = not reported. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustment were needed. The p-values presented here were reported in the 
original study. Means and standard deviations for the 27-state sample that is the focus of the study and this single study review were not included in the report, nor were they 
available from the authors because of data restrictions; the authors did, however, provide the number of charter students and schools for each analysis. 

This study is characterized as having a statistically significant positive effect on general reading achievement because the effect for at least one measure within the domain is positive 
and statistically significant, and no effects are negative and statistically significant. This study is characterized as having indeterminate effects on general math achievement because 
no estimated effects within the domain were statistically significant. For more information, please refer to the WWC Standards and Procedures Handbook, version 2.1 , page 96. 
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Appendix D: Supplemental findings by domain 


Mean 

(standard deviation) WWC calculations 


Domain and 
outcome measure 

Study 

sample 

Sample 

size 

Intervention 

group 

Comparison 

group 

Mean 

difference 

Effect 

size 

Improvement 

index 

p-value 

General reading achievement 








Gain in standardized 
reading scores 

27 states, 
Black students 

4,549 charter 
schools/ 
464,314 charter 
school students 

nr 

nr 

0.02 

0.02 

+1 

<0.01 

Gain in standardized 
reading scores 

27 states, 
Hispanic students 

4,549 charter 
schools/ 
561,155 charter 
school students 

nr 

nr 

0.00 

0.00 

0 

>0.05 

Gain in standardized 
reading scores 

27 states, 
White students 

4,549 charter 
schools/ 
624,914 charter 
school students 

nr 

nr 

-0.02 

-0.02 

-1 

<0.01 

Gain in standardized 
reading scores 

27 states, 
Asian students 

4,549 charter 
schools/ 
60,783 charter 
school students 

nr 

nr 

-0.01 

-0.01 

0 

<0.01 

Gain in standardized 
reading scores 

27 states, students 
receiving free/ 
reduced-price lunch 

4,549 charter 
schools/ 
974,981 charter 
school students 

nr 

nr 

0.02 

0.02 

+1 

<0.01 

Gain in standardized 
reading scores 

27 states, English 
language learners 

4,549 charter 
schools/ 
112,007 charter 
school students 

nr 

nr 

0.05 

0.05 

+2 

<0.01 

Gain in standardized 
reading scores 

27 states, students 
receiving special 
education services 

4,549 charter 
schools/ 
60,783 charter 
school students 

nr 

nr 

0.01 

0.01 

0 

<0.01 

Gain in standardized 
reading scores 

16 states, all charter 
schools 

3,619 charter 
schools/ 

1,425,630 charter 
school students 

-0.02 

(0.71) 

-0.02 

(0.48) 

0.01 

0.01 

+1 

<0.01 

Gain in standardized 
reading scores 

16 states, charter 
schools in the 2009 
study 

2,459 charter 
schools/ 

1,198,974 charter 
school students 

-0.02 

(0.70) 

-0.02 

(0.48) 

0.01 

0.01 

+1 

<0.01 

Gain in standardized 
reading scores 

16 states, new 
charter schools 
since the 2009 
study 

1,160 charter 
schools/226,656 
charter school 
students 

-0.03 

(0.71) 

-0.02 

(0.48) 

-0.01 

-0.01 

-1 

<0.01 


(continued) 
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General math achievement 


Gain in standardized 
math scores 

27 states, 
Black students 

4,547 charter 
schools/ 
456,470 charter 
school students 

nr 

nr 

0.02 

0.02 

+1 

>0.05 

Gain in standardized 
math scores 

27 states, 
Hispanic students 

4,547 charter 
schools/ 
531,183 charter 
school students 

nr 

nr 

-0.01 

-0.01 

0 

>0.05 

Gain in standardized 
math scores 

27 states, 
White students 

4,547 charter 
schools/ 
605,728 charter 
school students 

nr 

nr 

-0.07 

-0.07 

-3 

<0.01 

Gain in standardized 
math scores 

27 states, 
Asian students 

4,547 charter 
schools/ 
53,488 charter 
school students 

nr 

nr 

-0.04 

-0.04 

-2 

<0.05 

Gain in standardized 
math scores 

27 states, students 
receiving free/ 
reduced-price lunch 

4,547 charter 
schools/ 
937,554 charter 
school students 

nr 

nr 

0.03 

0.03 

+2 

<0.01 

Gain in standardized 
math scores 

27 states, English 
language learners 

4,547 charter 
schools/ 
102,095 charter 
school students 

nr 

nr 

0.05 

0.05 

+2 

<0.01 

Gain in standardized 
math scores 

27 states, students 
receiving special 
education services 

4,547 charter 
schools/ 
102,291 charter 
school students 

nr 

nr 

0.02 

0.02 

+1 

<0.01 

Gain in standardized 
math scores 

16 states, all 
charter schools 

3,582 charter 
schools/ 

1,348,562 charter 
school students 

-0.04 

(0.68) 

-0.03 

(0.46) 

-0.01 

-0.01 

-1 

<0.01 

Gain in standardized 
math scores 

16 states, charter 
schools in the 
2009 study 

2,425 charter 
schools/ 

1,129,199 charter 
school students 

-0.03 

(0.68) 

-0.03 

(0.46) 

-0.01 

-0.01 

-1 

<0.01 

Gain in standardized 
math scores 

16 states, new 
charter schools 
since the 2009 
study 

1,157 charter 
schools/219,363 
charter school 
students 

-0.05 

(0.69) 

-0.02 

(0.46) 

-0.03 

-0.03 

-2 

<0.01 


Table Notes: The supplemental findings presented in this table are additional findings that do not factor into the determination of the evidence rating. For mean difference, effect 
size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison group. The effect size is a 
standardized measure of the effect of an intervention on student outcomes, representing the average change expected for all students who are given the intervention (measured in 
standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an average student's percentile rank 
that can be expected if the student is given the intervention. Because this study examined year-to-year gains in achievement, the effect sizes and improvement indices presented 
are not directly comparable to effect sizes that would be estimated in an analysis of achievement levels, nr = not reported. 

Study Notes: No corrections for clustering or multiple comparisons and no difference-in-differences adjustment were needed. The p-values presented here were reported in the 
original study. The means and standard deviations for the 1 6-state sample were provided to the WWC by the authors. Means and standard deviations for the 27-state sample 
that is the focus of the study and this single study review were not included in the report, nor were they available from the authors because of data restrictions; the authors did, 
however, provide the number of charter students and schools for each analysis. 
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Endnotes 

1 1 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors]) to assess whether the study design meets WWC evidence standards. The review reports the WWC's assessment of whether 
the study meets WWC evidence standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol, version 2.0. A quick review of this study was released 
on July 30, 2013, and this report is the follow-up review that replaces that initial assessment. The WWC rating applies only to the 
results that were eligible under this topic area and met WWC standards without reservations or met WWC standards with reservations, 
and not necessarily to all results presented in the study. The review focused on analyses that were emphasized in the study and analy- 
ses that examined key student subgroups. The study presented results from analyses, including those based on different numbers of 
school years in the data, by state, by school grade span (elementary, middle, high), and by attendance in a charter school operated by 
a charter management organization. 

2 The WWC also reviewed the evidence from the earlier study that focused on a subset of 16 states: 
http://ies.ed.gov/nceeA/VWC/pdf/quick_reviews/charterschools_021710.pdf 

Recommended Citation 

U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse. (2014, January), l/l/l/l /C 
review of the report: National Charter School Study: 2013. Retrieved from http://whatworks.ed.gov 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of students, the improvement index represents the gain 
or loss of the average student due to the intervention. As the average student starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which subjects are assigned 
to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which investigators randomly assign 
eligible participants into intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% Ip < 0.05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 2.1) for additional details. 
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